Computer vision tasks often have side information available that is helpful to solve the task. For example, for crowd counting, the camera perspective (e.g., camera angle and height) gives a clue about the appearance and scale of people in the scene. While side information has been shown to be useful for counting systems using traditional hand-crafted features, it has not been fully utilized in counting systems based on deep learning. In order to incorporate the available side information, we propose an adaptive convolutional neural network (ACNN), where the convolution filter weights adapt to the current scene context via the side information.

Summary of the Paper: This work proposes to use adaptive convolutions (also called'cross convolutions') to incorporate side information (e.g., camera angle) into CNN architectures for vision tasks (e.g., crowd counting). The filter weights in each adaptive convolution layer are predicted using a separate neural network (one network for each set of filter weights) with is a multi-layer perceptron. This network is referred to as'Filter Manifold Network' which takes the auxiliary side information as input and predicts the filter weights. Experiments on three vision tasks of crowd counting, digit recognition and image deconvolution indicate the potential of the proposed technique for incorporating auxiliary information. In addition, this paper contributes a new dataset for crowd counting with different camera heights and angles.

Computer vision tasks often have side information available that is helpful to solve the task. For example, for crowd counting, the camera perspective (e.g., camera angle and height) gives a clue about the appearance and scale of people in the scene. While side information has been shown to be useful for counting systems using traditional hand-crafted features, it has not been fully utilized in counting systems based on deep learning. In order to incorporate the available side information, we propose an adaptive convolutional neural network (ACNN), where the convolution filter weights adapt to the current scene context via the side information. The filter weights are generated using a learned filter manifold'' sub-network, whose input is the side information.

Probabilistic matrix factorization (PMF) is a powerful method for modeling data associated with pairwise relationships, finding use in collaborative filtering, computational biology, and document analysis, among other areas. In many domains, there is additional information that can assist in prediction. For example, when modeling movie ratings, we might know when the rating occurred, where the user lives, or what actors appear in the movie. It is difficult, however, to incorporate this side information into the PMF model. We propose a framework for incorporating side information by coupling together multiple PMF problems via Gaussian process priors. We replace scalar latent features with functions that vary over the space of side information. The GP priors on these functions require them to vary smoothly and share information. We successfully use this new method to predict the scores of professional basketball games, where side information about the venue and date of the game are relevant for the outcome.